MODULATION SPACES OF SYMBOLS FOR 
REPRESENTATIONS OF NILPOTENT LIE GROUPS 
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Abstract. We investigate continuity properties of operators obtained as val- 
ues of the Weyl correspondence constructed by N.V. Pedersen (Invent. Math. 
118 (1994), 1-36) for arbitrary irreducible representations of nilpotent Lie 
groups. To this end we introduce modulation spaces for such representations 
and establish some of their basic properties. The situation of square inte- 
grable representations is particularly important and in the special case of the 
Schrodinger representation of the Heisenberg group we recover the classical 
modulation spaces used in the time-frequency analysis. 



1. Introduction 

The representation theory of the (2n -I- l)-dimensional Heisenberg group ]Hl2„+i 
provides a natural background for the pseudo-differential calculus on R". It is well 
known that the representation theoretic approach has led to a deeper understanding 
of the Weyl calculus, which resulted in simplified proofs and improvements for many 
basic results. A celebrated example in this connection is the Calderon-Vaillancourt 
theorem on L^-boundedness for pseudo-differential operators ( [CV72| ). This classi- 
cal theorem was strengthened in the paper |GH99] by using the modulation spaces, 
which are function (or distribution) spaces defined in terms of the Schrodinger rep- 
resentations of Heisenberg groups. The modulation spaces were introduced in |Fe83| 
in the framework of harmonic analysis of locally compact abelian groups. 

On the other hand, a remarkable Weyl calculus was set up in [Pe94j for arbitrary 
unitary irreducible representations of any nilpotent Lie group. We shall call it the 
Weyl-Pedersen calculus. It is a challenging task to understand this interaction of 
the ideas of pseudo-differential calculus with the representation theory of nilpotent 
Lie groups. 

In the present paper we address the above problem in the shape of the L^-bound- 
edness theorems. Specifically, we are going to investigate continuity properties of 
the operators constructed by the Weyl-Pedersen calculus. For this purpose we in- 
troduce the modulation spaces M^''^{tt) defined in terms of an arbitrary irreducible 
representation tt of a nilpotent Lie group G. One key feature of our representation 
theoretic approach is that if O stands for the coadjoint orbit corresponding to tt 
f [Kir62j ). then the symbols of the operators constructed by the Weyl-Pedersen cal- 
culus are functions or distributions on the coadjoint orbit O, while the Hilbert space 
LP'{0) carries a natural irreducible representation tt'^ of the nilpotent Lie group 
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G tK G. Therefore our general notion of modulation spaces for irreducible represen- 
tations allows us to investigate the modulation spaces of symbols for the operators 
constructed by the Weyl-Pedersen calculus for the representation tt. This approach 
also reveals the representation theoretic background of the i^-boundedness theorem 
of |GH99| . 

We find several of the familiar properties of the classical modulation spaces, such 

as: 

- continuity of the operators constructed by the Weyl-Pedersen calculus with 
symbols in an appropriate modulation space M^'^(7r^) fCorollarv l2.26p : 

- independence on the choice of a window function, and covariance of the 
Weyl-Pedersen calculus, in the case of square-integrable representations 
(Theorems and ES]) . 

Besides the aforementioned reasons, the present research has also been motivated 
by the recent interest in the magnetic pseudo-differential Weyl calculus on R" (see 
for instance [MP 04] . |IMP07| . [MP09| . and the references therein), which was par- 
tially extended to nilpotent Lie groups in the papers |BB09a) and [BB09b| . Specif- 
ically, the results of the present paper apply to the Weyl calculus associated with a 
polynomial magnetic field on R", in particular complementing the L^-boundedness 
theorem established in [IMP07j for magnetic fields whose components are bounded 
and so are also their partial derivatives of arbitrarily high degree. 

Notation. Throughout the paper we denote by S{V) the Schwartz space on a 
finite-dimensional real vector space V. That is, S{V) is the set of all smooth func- 
tions that decay faster than any polynomial together with their partial derivatives 
of arbitrary order. Its topological dual — the space of tempered distributions on 
V— is denoted by 5' (V). 

We shall also use the convention that the Lie groups are denoted by upper case 
Latin letters and the Lie algebras are denoted by the corresponding lower case 
Gothic letters. 

For basic notions on Weyl pseudo-differential calculus, we refer to jHor07| . [Fo89| 
and [G?nT] . 

2. Modulation spaces for unitary irreducible representations 
2.1. Preliminaries on semidirect products. 

Definition 2.1. Let Gi and G2 be connected Lie groups and assume that we have 
a continuous group homomorphism a: Gi ^ AutG2, gi <—>■ ag-^. The correspond- 
ing semidirect product of Lie groups Gi Kq, G2 is the connected Lie group whose 
underlying manifold is the Cartesian product Gi x G2 and whose group operation 
is given by 

(.91,52) • {hi,h2) = (51/11, 0^-1(52)^12) (2.1) 

whenever 5j, /ij € Gj for j — 1,2. 

Let us denote by d: gi ^ Derg2 the homomorphism of Lie algebras defined as 
the differential of the Lie group homomorphism Gi Aut02, gi ^ ^(agj). Then 
the semidirect product of Lie algebras Qi Kq, 52 is the Lie algebra whose underlying 
linear space is the Cartesian product fli x 02 with the Lie bracket given by 

[{Xi,X2),{Yi,Y2)] - ([Xi,ri],d(Xi)r2 -d(yi)X2 + [X2,Y2]) (2.2) 
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if Xj, Yj G 2j foi' 3 — I7 2. One can prove that gi Kq. 32 is the Lie algebra of the Lie 
group Gi Kq 62 (see for instance Ch. 9 in [Ho65j ). □ 

Remark 2.2. Let Gi and G2 be nilpotent Lie groups and a: Gi ^ AutG2 be a 
unipotent automorphism, in the sense that for every Xi e gi there exists an integer 
m > 1 such that — 0. Then an inspection of (|2.2p shows that gi 92 is 

a nilpotent Lie algebra, hence Gi Xq G2 is a nilpotent Lie group. □ 

Example 2.3. For an arbitrary Lie group G with the center Z, let us specialize 
Definition for Gi — G2 ■= G and a: G ^ AutG, g 1-^ ag, where C(g{h) = ghg~^ 
whenever g,h G G. Then the corresponding semidirect product will always be 
denoted simply by G k G and has the following properties: 

(1) If G is nilpotent, then so is G k G. 

(2) The Lie algebra of G k G is g Xadg 0, which will be denoted simply by g k g, 
and the center of G k G is Z x Z. 

(3) The exponential map of the Lie group G k G is given by 

exPGKG(^>^) = (expG^,expg(-X)expG(X + F)) 

for every {X, F) £ g Kad^ g. 

(4) The mapping 

fi:GxG^G^G, {g,h)^{gh,g) 

is an isomorphism of Lie groups, and the corresponding isomorphism of Lie 
algebras is L(^) : g x g g k g, {X, Y) {X + Y, X). 

In fact, property llj follows by Remark 12.21 Property 12]) is a consequence of the 
fact that a = adg : g — > Der g along with (|2.ip . 

To prove property ([3]), note that the mapping 11: G ix G — > G, (gi, 52) — > 5i52 is 
a homomorphism of Lie groups, hence we have the commutative diagram 

L(n) 



S 



expe 



G K G > G 

n 

where it is easy to see that the Lie algebra homomorphism L(n) : g k g — > g is given 
by {X, Y) X + Y. Now let {X, F) G g x g arbitrary. It is clear that there exists 
g ^ G such that bxjpq^q^X, Y) = (exp^ X, g), and then the above commutative di- 
agram shows that expQ{X + Y) = Il{expQ^Q{X,Y)) = Il{expQ X, g) = {expQX)g, 
whence g = expQ{—X) expQ{X + Y). 

Finally, property ([4]) follows by a straightforward computation. □ 

2.2. Weyl-Pedersen calculus for unitary irreducible representations. 

Setting 2.4. Throughout the present section we shall use the following notation: 

(1) Let G be a connected, simply connected, nilpotent Lie group with the Lie 
algebra g. Then the exponential map exp^ : g ^ G is a diffeomorphism 
with the inverse denoted by logg : G ^ g. 

(2) We denote by g* the linear dual space to g and by (•,•): g* x g ^ K the 
natural duality pairing. 

(3) Let ^0 S 3* with the corresponding coadjoint orbit O :— AdQ(G)^o Q Q* ■ 
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(4) The isotropy group at is G^o := {g G G \ AdQ(g)^o = Co} with the 
corresponding isotropy Lie algebra g^g = {X G g | Co o adgX = 0}. The 
center 3 := {X G g | [X, g] = {0}} clearly satisfies 3 ^ g^o- 

(5) Let n := dimg and fix a sequence of ideals in g, 

{0} = 00 C 01 C • • • C g„ g 

such that dim(0j/gj_i) — 1 and [g, gj] C gj_i for j = 1, . . . , n. 

(6) Pick any Xj e 0^- \ for j = 1, . . . , n, so that the set {Xi, . . . , X„} will 
be a Jordan-Holder basis in g. 

(7) The set of jump indices of the coadjoint orbit O with respect to the above 
Jordan-Holder basis is e := {j G {1, . . . ,n} | 0j 2 0i-i + 04o} ^''^'^ "^o^s 
not depend on the choice of Co G O (see also Prop. 2.4.1 in |Pe84] ) . The 
corresponding predual of the coadjoint orbit O is 

0e := span{Xj | j G J^^} C g. 

We shall denote e = {ji, . . . , jd} with 1 < ji < ■ ■ ■ < jd < n. 

(8) We shall always consider O endowed with its canonical Liouville measure 
(see for instance the remark after the statement of the theorem in § 6, Ch. 
II, Part 2 in [Pll67) V 

(9) Let n: G ^ ^i'H) be a fixed unitary irreducible representation associated 
with the coadjoint orbit O by Kirillov's theorem f [Kir62j ). 

□ 

Remark 2.5. The space of smooth vectors Hoo {v G Ti, \ Tr{-)v G C°°{G, H)} is a 
Frechet space in a natural way and is a dense linear subspace of Ti which is invariant 
under the unitary operator T:{g) for every g G G. The derivate representation 
d7r: g — ^ End (Hoc) is a homomorphism of Lie algebras defined by 

(VX G g, f G Hoo) dn{X)v = ^ 7r{expa{tX))v. 

at t=o 

We denote by Ti-^oa the space of all continuous antilinear functionals on Tioo and 
the corresponding pairing will be denoted by (• | •) : 7i_oo x Woo ^ C. just as 
the scalar product in 7i, since they agree on Tioo x Hoo if we think of the natural 
inclusions Hoo ^ H ^ H-oo- (See for instance |Ca76| for more details.) □ 

Remark 2.6. We now recall a few facts from subsect. 1.2 in |Pe94j for later use. 
Let us denote by 6p{H) the Schatten ideals of operators on H for 1 < p < 00. 
Consider the unitary representation tt®^ : G x G ^ B{62{H)) defined by 

(Vgi,52 G G)(Vr G 62(H)) 7T^'{gi,g2)T = n{g,)Tn{g2)'' . 

It is not difficult to see that tt®^ is strongly continuous. The corresponding space 
of smooth vectors is denoted by B{H)oo and is called the space of smooth operators 
for the representation tt. One can prove that actually B{H)oo ^ &i{H)- 

For an alternative description oiB{H)oo let gc := gi^RC be the complexification 
of g with the corresponding universal associative enveloping algebra U(gc). Then 
the aforementioned homomorphism of Lie algebras d7r has a unique extension to 
a homomorphism of unital associative algebras d7r: U(gc) — > End(7ioo)- One can 
prove that for T G B{H) we have T G B{7{)oo if and only if T[H) + T*[H) C H^ 
and d7r(M)T,d7r(M)r* G B{H) for every u G U(0c). 
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Since {(• I /i)/2 I /i,/2 e Hoo} C B{n)oo C 61(H) and Hoc is dense in H, we 
get continuous inclusion maps 

B{n)oo ^ &i{n) ^ B{n) ^ B{ny^, (2.3) 

where the latter mapping is constructed by using the well-known isomorphism 
(61 (7i))* ~ BCH) given by the usual semifinite trace on B{H). □ 

Definition 2.7. The Fourier transform S{0) S{ge), a i^a, defined by 

aiX) = J e-'<«'^>a(OdC 
o 

is an isomorphism of Frechet spaces. The Lebesgue measure on ge can be normalized 
such that the Fourier transform extends to a unitary operator 

and its inverse is defined by the usual formula (see Lemma 4.1.1. in |Pe94j ). We 
shall always consider the predual ge endowed with this normalized measure. 

If / G 'H-oo and <j) £ Ti-oo, or /, (/) e 7Y, then we define the corresponding 
ambiguity function 

A{fA)^A^f: 0e ^C, {A^f)[X) ^ if I niexpaX^). 

For (j) G H-oo and / G Hoo we also define {A^f){X) ~ {(f) \ TT{expQ{—X))f) when- 
ever X e fle- 

It follows by Proposition ESUI) below that if f,(j) e TL, then A^^f € ^^(fle), 
so we can use the aforementioned Fourier transform to define the corresponding 

cross-Wigner distribution yV(/, (j)) G L'^iO) such that >V(/, 0) := .A^/. □ 

The second equality in Proposition l2.8t [T|) below could be referred to as the 
Moyal identity since that classical identity (see for instance |Gr01j ) is recovered in 
the special case when G is a simply connected Heisenberg group. 

Proposition 2.8. The following assertions hold: 

(1) If(f)eH, then A4,f G L'^{Qc)- We have 

{A^Ji I A^,J2)l^s,) = (/i I h)n ■ {(t>2 I 

= {Wif,,^i)\Wif2,h))LHO) ■ 

for arbitrary 0i, 02, /i, /2 G 

(2) If (j)o G H with ||(/>o|| = 1, then the operator .A^^ : H L^(0e), / >— > A^igf, 
is an isometry and we have 

J {A^J){X)-7T{expaX)(l)dX = (0 | 0o)/ 

Be 

for every (j) G Tioo o,nd f d Ti.. In particular, 

J {A^J){X)-7T{expaX)<PodX = f 

Be 

for arbitrary / G 
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Proof. (P) We first prove that jM]) holds for 0i, ?!)2, /i, /2 £ Woo- Since S(Hoo) is 
contained in the ideal 6i{H) of trace-class operators, it makes sense to define 

{VAeB{n)oc) f^.G^C, f^{x)=Tr{7T{x)A). 

It follows by Th. 2.2.7 in [Pe94] that for the suitably normalized Lebeseue measure 
on 0e we have for every A,B& B{TL)oo, 

J f^{expaX)fB{e^p^X)dX^TT{AB*). (2.5) 

Be 

We now denote 

(V/,0eH) A/,^ = (• I 0)/ e 6(H) 
and recall that for arbitrary /, /i, /2, 0, '/'i, '/'2 G 'H we have 

^*f,<p = ^0,/, Tr(A/,0) = (/ I 0), and Aj^^^^Af^^^^ = A/i, (011/2)02 = Af2\4'i)fi.4>2- 

It then easily follows that if /, G "Hoc, then Af^^ e B(H)oc and for arbitrary 
X G fle we have 

/^^'^(expGX) - Tr(7r(expGX)A/,^) = Tr (A,(,,p^ x)/,^) = (^(exp^X)/ | 0) 
= (/ I 7r{expa{-X))cj,), 

whence 

(VXege) f^'''{expaX)^iA^f)i-X). (2.6) 

Now, by using (12. 5p for A := and i? := A/^^^^, we get 

(yl0i/i I ^02/2)^2(5^) = Tr(^/i^0iA}^ = Tr(A/i,0i^02jJ = Tr (Ajj_(0^|0,)/J 

= (,/i I (01 I h)f2) = ifi I /2)h • I 0i)h 

The second part of (|2.4|) then follows since the Fourier transform L^{0) — > i^(0e) 
is a unitary operator, as we already mentioned in Definition 12.71 

The extension of (|2.4p from Tioo to H proceeds by a density argument. First 
note that by (|2.4p for — (j)2 ~: (j) & Hoo and /i = /2 =: / £ "Woo we get 
1 1 -^0/1 1 — ll'/'ll ' 11/11- Since Hoo is dense in 7i, it then follows that the sesquilinear 
mapping Hoc x Hoo 'H, (/, </>) 1-^ extends uniquely to a sesquilinear mapping 
7i X H — > H satisfying 

(V/,0£7i) (2.7) 

Now the first part of (|2.4p follows as a polarization of (|2.7p . and then the second 
part follows by using the Fourier transform L^{0) — > L^{Qe) as above. 

([2]) It follows at once by Assertion ^ that the operator A^,^ : TL — > -/^^(ge) is 
an isometry if ||(/'o|| = 1- The other properties then follow immediately; see for 
instance Proposition 2.11 in [Fii05| . □ 

We now draw some useful consequences of Proposition 12.81 We emphasize that 
Assertion ^ in the following corollary in the special case of square-integrable rep- 
resentations reduces to a theorem of |Co84] and |CM96] . One thus recovers Th. 2.3 
in [GZOlj in the case of the Schrodinger representation of the Heisenberg group. 

Corollary 2.9. If 4>o G 'Hoo with — 1, then the following assertions hold: 
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(1) For every f G H-oo we have 

JiA^J)iX)-Tr{e^PaX)<PodX = f (2.8) 

Be 

where the integral is convergent in the weak* -topology o/7i_oo. 

(2) If f E Ti-oc, then the above integral converges in the Frechet topology ofTLao- 

(3) If f E H-oo, then we have f G Hoc if and only if A,f,gf G S{Qe)- 

Proof. If / e H-oo, we have to prove that J {Acf,gf){X) ■ (7r(expQ X)0o | (j)) dX = 

Be 

(/ I 0), for every (j) £ Hqo, that is, 

J if I 7r(expGX)0o) ■ (7r(expGX),^o | <^) = (/ | 0). 

Be 

Since (/ | •) is an antihnear continuous functional, the above equation will follow 
as soon as we have proved that for (j) G Woo we have 

J {(j) I 7r(expQ X)(/)o)7r(expc X)(/)o dX = (j) 

Be 

with an integral that converges in the topology of Tioo ■ Note that this is precisely 
Assertion To prove it, we just have to use Proposition l2.8t P|) along with the fact 
that for (/),(/>o G T-Lao the function X ^ [(f) \ 7r(expQ X)(/)o) = {A^„(f)){X) belongs 
to S{Qe) (see Th. 2.2.6 in [Pe94| ) while the function X ^ 7r(expQX)0o and all its 
partial derivatives have polynomial growth. 

For Assertion we have just noted that if / G H-oo then -40,,/ G S{Qe) as a 
direct consequence of Th. 2.2.6 in |Pe94j . Conversely, if / G H-oo has the property 
Acftgf G 5(ge), then the fact that all the partial derivatives of X i~> 7r(exp(3X)0o 
have polynomial growth implies at once that the integral in (|2.8p is convergent in 
the Frechet space Tiooi hence Assertion ([1]) shows that actually / G Ti-oo- D 

Definition 2.10. The Weyl-Pedersen calculus Op'^(-) for the unitary representa- 
tion TT is defined for every a G S{0) by 

0])^{a) = j a(X)7r(expGA:)dA: G S(H). (2.9) 

Be 

This definition can be extended to an arbitrary tempered distribution a G S'{0) 
by using Th. 4.1.4 and Th. 2.2.7 in |Pe94| to define an unbounded operator Op'^(a) 
such that 

(V6 G S{0)) Tr (0p'^(a)0p'^(6)) = (a, 6), (2.10) 

where we recall that (•, •) : S'{0) x S{0) C stands for the usual pairing between 
the tempered distributions and the Schwartz functions. We say that a G S'{0) is 
the symbol of the operator Op'^(a). □ 

We now record some basic properties of the Weyl-Pedersen calculus constructed 
in Definition 12. 101 These are actually direct consequences of Proposition l2.8p ]l. 



Corollary 2.11. The following assertions hold: 
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(1) For each a G 5(0) we have 

(Op^(a)0 I f)n = (a | = (« I 

whenever (/), f G Ti.. Similar equalities hold if a £ S'{0) and (f), f d TLoo- 

(2) //(/)i,02 G "^oo a'^'^ a := W(0i,(/)2) £ 5(0), then Op^(a) is a rank-one 
operator, namely Op'^(a) = (• | (f>2)<pi- 

Proof. Assertion ([IJ is a consequence of formula (|2.9p along with Definition 12.71 
Then Assertion ([2]) follows by Assertion ^ along with Proposition 12. 8p ]l. In fact, 

(Op-(>V(0i,02))/ I 0) = (>V(0i,(/.2) I W(<^,/)) = (01 I </)) • (/ I </)2) 
= ((/ I 02)<^1 I 4>) 

for arbitrary (j) G H. □ 

Remark 2.12. We can define the cross- Wigner distribution W{fi, /2) G ^'(0) for 
arbitrary /i, /2 £ H-oo as follows. An application of Th. 1.3(b) in [Ca76| shows that 
if j4 G B{H)oo and / G H-oo, then Af G Hqo, in the sense that there exists a smooth 
vector denoted Af such that for every G Hoc we have (/ | A*(j)) = {Af \ 0). 
Moreover, we thus get a continuous linear map A: TL-oo — > Hoo whose restriction 
to Ti is the original operator A G B{H)oo- Then for /i, /2 G 7i-oo we can define 
the continuous antilinear functional 

Tf,j, : B{H)oo -> C, Tf,j, {A) := (A | Af^). 

That is, Tf^j^ G B{n)*^, and then Th. 4.1.4(5) in [Pe94) shows that there exists a 
unique distribution a/1,/2 G '5'(0) such that Op^ (uf-^j^) = Tf^j^. Now define 

W(/i,/2) 

We can consider the rank-one operator Sf^j^ := (• | f2)fi- Hoo H-oc and for 
arbitrary A G B{TL)oo thought of as a continuous linear map A: TL-00 ^ Woo as 
above we have 

TT{Sf,j,A) ^ {f^ \ Af2) ^Tf.jM)- 
Thus the trace duality pairing allows us to identify the functional T/i j2 G B{T-L)*^ 
with the rank-one operator (• | /2)/i, and then we can write 

(V/1,/2 G U-oo) Op'^(W(/l,/2)) = (• I /2)/l. (2.11) 

In particular, it follows that the above extension of the cross- Wigner distribution 
to a mapping yV(-, ■) : 7i-oo x 'H-oo S'{0) allows us to generalize the assertion 
of Corollarv [2lTl[2|l to arbitrary (/>i, 02 G H-oo- □ 

Definition 2.13. Recall from Th. 4.1.4(5) in |Pe94j that the Weyl-Pedersen calcu- 
lus Op'^ : S'(0) B{TL)l^ is a linear isomorphism and a weak*-homeomorphism. 
We introduce the linear space 

S°{0) := {a G S'{0) \ Op" (a) G B{n)} 

(see (|2.3p ). Then the mapping Op'^ induces a linear isomorphism 5^(0) B{Ti,), 
hence there exists an uniquely defined bilinear associative Moyal product 

S^{0)y.S^{0)->S^{0), {a,b)^ai^b 

such that 

(Va, b G 5°(0)) Op"(a#&) = Op" (a) Op" (6). 
The space of distributions 5^(0) is thus made into a VF*-algebra such that the 
mapping S'^{0) B{7i), a ^ Op" (a) is a ^-isomorphism. □ 
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With Dcfinition l2.13l at hand, we can say that one of the main problems addressed 
in the present paper is to describe large classes of distributions belonging in the 
space 5"(0). 

Example 2.14. Here are some examples of distributions in S^{0) which are al- 
ready available. 

(1) It follows at once by (|2^ and (|2?T0l) that 

{ae5'(0) |aeii(0e)} C5"(0). 

(2) The Schwartz space S{0) is a *-subalgebra of 5^(0) and the mapping 
Op'' : S{0) B{H)oo is an algebra ^-isomorphism by Th. 4.1.4 in [Pe94j . 

(3) The space L^{0) is a *-subalgebra of 5"(C'), and Op^: L^{0) ^ 62(H) is 
a unitary operator and an algebra *-isomorphism as an easy consequence 
of Th. 4.1.4 in [Pe94j : see also |Ma07j . 

(4) For every Y £ q we have e'^''^^ S S^{0) since it follows at once by (|2.9|) 
and ((2TU)) that Op''(e'<-^'^>) = TT{expaY). 

See also Corollary d^i for the important example M^''^{tt*) ^ S°{0). □ 
2.3. Modulation spaces. 

Definition 2.15. Let 4> G Woo \ {0} be fixed and assume that we have a direct 
sum decomposition ge = qI + qI- 

Then let 1 < r, s < 00 and for arbitrary / e H-oo define 

II/IIa/;- = (/(/ \{A^f)iXi,X2Wdx^y^'dX2y^' e [o,oo] 

si si 

with the usual conventions if r or s is infinite. Then we call the space 

Ml'^in) := {/ £ I < ^} 

a modulation space for the irreducible unitary representation n: G B{7i) with 
respect to the decomposition — qI'x 0e the window vector (j) e Tioa \ {0}. □ 

Remark 2.16. Assume the setting of Definition 12.151 and recall the mixed-norm 
space L''''^{2l x g^) consisting of the (equivalence classes of) Lebesgue measurable 
functions 6 : g^ x g^ ^ C such that 




(cf. \Gm\ ). It is clear that Ar/in) = {/ e H-oc \ A^f G L^''{qI x qI)}. □ 
Example 2.17. For any choice of e Tioo \ {0} in Definition l2.15l we have 

Indeed, this equality holds since ||-40/||i2(g_^) = ■ \\f\\ for every f & H (see (|2.7p 
in the proof of Proposition 12.81 above). □ 
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2.4. Continuity of Weyl-Pedersen calculus on modulation spaces. In the 

following lemma we use notation introduced in Example l2.3l| 4|) and Remark 12.61 

Lemma 2.18. Let G he any Lie group with a unitary irreducible representation 
■n: G ^ B{Ti.) and define 

tt-.GkG^ B{e2{n)), TT{g, h)T = 7r(g/i)r7r(g)-i. 

Then the following assertions hold: 

(1) The diagram 

G K G — ^^(62(7^)) 
GxG 

is commutative and n is a unitary irreducible representation of G \>i G. 

(2) The space of smooth vectors for the representation n is BiTOjoo- 

(3) Let us denote by g = q tK g and define 

^ |(Xj_„,Xj_„) for j ^n+l,...,2n. 

Then Xi, . . . , X2n is a Jordan-Holder basis in g and the corresponding pre- 
dual for the coadjoint orbit O C g* associated with the representation tt 
is 

0e = 0e X Be ^ Q, 

where e is the set of jump indices for O. 

Proof. ([T]) It is clear that the diagram is commutative, and then the mapping tt is 
a representation since tt®^ is a representation and fi: GxC^CkG is a group 
isomorphism. It is well-known that the representation tt*^^ is irreducible, hence n 
is irreducible as well. For the sake of completeness, we recall the corresponding 
reasoning. Let arbitrary A G B{&2{H)) satisfying 

(V(g, /i) e G K G) A^{g,h)^^{g,h)A. (2.12) 

We have to show that ^ is a scalar multiple of the identity operator on 62(H). For 
that purpose, let us define the operators is, Rb ■ &2{'H) — > &2{'H) by LbX = BX 
and RbX = XB for X,B e B{H). Note that if G G, then 7f(l,/i) = 
It then follows by (|2.12p that — Lt^(ji-^A for every h ^ G. On the other 

hand, the representation tt is irreducible, the linear space span{7r(/i) | h £ G} 
is dense in B{T-C) in the strong operator topology, and then it easily follows that 
ALb = Lb A for every B G B{TL). This property implies that there exists A G B{Ti) 
such that A — Ra (see for instance |Ta03| ) . Now, by using (|2.12p for h = 1, we 
get TT{g)XTr{g)~^A — Tr{g)XATT{g)~^ for every g € G and X G B{TC)^ which implies 
^■'''(5) = ''^{9)-^ for arbitrary g £ G. Since tt is an irreducible representation, it 
follows that A is a scalar multiple of the identity operator on TC, hence A — La is 
a scalar multiple of the identity operator on 62(7^), as we wished for. 

^ This assertion follows by Remark 12.61 

([3]) It is easy to see that the sequence 

(0,Xi),...,(0,X„),(Xi,0),...,(X„,0) 
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is a Jordan-Holder basis in the direct product x g, and the coadjoint orbit corre- 
sponding to the representation Tr®^ ; G x G ^ B{&2{'H)) is O x C (This follows 
for instance by the theorem in § 6, Ch. II, Part 2 in jPu67| .) Then the assertion 
follows by Example 12. 3t |4l) along with the above Assertion ([T|). □ 

In the following definition we use an idea similar to one used in jMP09] . 

Definition 2.19. Let G be a simply connected, nilpotent Lie group with a unitary 
irreducible representation tt: G ^ B{H). Assume that O C g* is the coadjoint 
orbit associated with this representation and define 

n*:G^G^BiL'{0)), 7r#(expG X, exp^ F)/ = e'<-'^>#e'<-^>#/#e-'<-^> , 

where # is the Moyal product associated with tt (see Definition [2Tl3]). We note the 
following equivalent expression 

(VX,re0) ^#(expG,G(^,n)/-e'^^+^)#/#e-<-'^> (2.13) 

which follows by Example I2.3[[3ll . The corresponding ambiguity function is given 

by 

AtF: 0e X fle C, iAtF)iX,Y) = {F \ n* ie^pa>cGiX,Y))1>) 
for $, F G L^{0) or for a function $ G S{0) and a continuous antilinear functional 
F: 5(0) ^ C denoted by ^ (i^ I □ 

Remark 2.20. To explain the terminology of Definition I2.19i let us see that we 
really have to do with the ambiguity function of a unitary representation. To this 
end, recall the unitary operator Op'^ : L'^{0) &2{'H) (see e.g., Example 12 . 14t| 3|) ) 
and the representation tt: G x G ^ B{&2{'H)) from Lemma [2.181 It follows by 
Definition 12.131 and Example I2.14t |^ that the unitary operator Op^ intertwines 
tt"^ and tt, hence we get by Lemma 12.181 that tt'^ is also a unitary irreducible 
representation. It also follows that ge x C g k g is a predual to the coadjoint 
orbit O'^ C (g K g)* associated with the representation tt^. 

Let us note that the space of smooth vectors for the representation tt"^ is equal to 
S{0), as a consequence of Lemma 12. 18p |). since Op'^ : S{0) — > B{TL)oo is a linear 
isomorphism by Th. 4.1.4 in [Pe94j . □ 

The next statement points out the representation theoretic background of the 
computation carried out in the proof of Lemma 14.5.1 in [GrOlJ. 

Proposition 2.21. Let G he a simply connected, nilpotent Lie group with a unitary 
irreducible representation tt: G — > B{Ti). Pick any predual ge ^ for the coadjoint 
orbit O Q g* corresponding to the representation tt. // either 4>i,4>2i fi, f2 G W or 
(/)i,(/>2 e T-Loo and /i,/2 £ K^oo, then 

{VX,Y G Be) A*{W{fij2)){X,Y) = {A^Ji){X + Y)-{A^j2){X) 

where $ :== >V((/)i,02) G i^(C), while W(-,-) and A4>Jf. ge ^ C for j = 1,2 
are cross-Wigner distributions and ambiguity functions for the representation n, 
respectively. 

Proof. If we denote F — yV(/i,/2), then for arbitrary X,Y G ge we have by 
Definition [231 Example [OP , and Remark [1201 

{A*F){X,Y) = {F I n*{expa.GiX,YmL2^o) 

= [F I ^#(expG X, (expG X)-' e^pai^ + 1'))$)l^(o) 
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= {Op^F) I ^(expG X, (expG X)"^ expa{X + Y))Op^^))e,iH) 
= (Op^F) I ^(expG(X + y))0p-($)7r(expGX)-i)e,(«). 
On the other hand Remark 12.121 fparticularlv (|2.1ip ) shows that 

Op^F) = (• I h)h 
and Op'^($) = (• I 02)01, whence 

7r(expG(X + r))Op"(<i>)^(expG X)-' = (• | ^(exp^ X)^2Hexpa{X + Y))^,. 
Then the above computation leads to the formula 

iAtF){X,Y) = (^(expGX)02 I /2) • ifi I ^(expG(X + r))0i), 
which is equivalent to the equation in the statement. □ 

We now prove a generalization of Th. 4.1 in [To04| to irreducible representations 
of nilpotent Lie groups. 

Theorem 2.22. Let G be a simply connected, nilpotent Lie group with a unitary 
irreducible representation tt: G — > B{'H). Let O be the corresponding coadjoint 
orbit, pick 01,02 G '^oo \ {0}, and denote $ — yV'(0i,02) G S{0). Assume that 
ge is a predual to the coadjoint orbit O, and let 0e = 0e "i" fle '^'^V direct sum 
decomposition. 

If 1 < r < s < oo and ri,r2,Si,S2 £ [r,s] satisfy J- + ^ = i + ^ = i + i, 
then the cross- Wigner distribution defines a continuous sesquilinear map 

W(.,.): M;;'-(7r) X M^'^W ^ M^^\n*). 

Proof. Let /i, /2 G H-oo and note that for every X S ge we have 

{A^j2){X) = (/2 |^(expGX)02) = {AMhX). (2.14) 

Therefore by Proposition 12.211 we get 

I|W(/i,/2)||m;-(.#) = (1 ^^(>^2)dy2)'^', (2.15) 
si 

where 

i^(i^2) =/(/ y"|(^0i/l)(^l+n,X2+>'2)-(^/.02)(-^l,-^2)rdXidX2)'^''dYi. 

BI fle Be 

(2.16) 

On the other hand, it follows by Minkowski's inequality that for every measurable 
function F : g^ x g^ x g^ ^ C and every real number t > 1 we have 

(y"(y" |r(ri,X2, Fa) ldX2)*dYi)'^* < J (^J \T{Yi,X2,Y2)\'dYiy^'dX2 (2.17) 

Be Be Be Be 

whenever Y2 £ g^. By and ((TTf)) with t := s/r and 

T{Yi,X2,Y2) := J \{A4,Ji){Yi~Xi,Y2-X2)-{Af,h)iXi,X2)rdXi 

Bl 
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we get 

rjs \ s/r 



si el 

=(| ||r(.,X2,y2)||L./.(Bi)dX2 



(2-18) 



Now note that T{-, X2,Y2) is equal to the convolution product of the functions 
\{A^i.fi){-,Y2 - X2)r and |(^/2(/)2)(-,X2)r- It follows by Young's inequality that 

||r(.,X2,r2)IL./.(Bi) < \\\iA^ji){-,Y2 ^ X2WhH^,iy \\\iAfMi-,X2W\\L^H9i) 

whenever ti,t2 S [1, oo] satisfy + — 1 + f - By using the above inequality with 
= ^ for j = 1, 2, and taking into account (|2.18p . we get 

F(Y2)<([ ||M0,/l)(-,r2-X2)|l2-l(gi)IIM/.02)(-,X2)|IU(8MdX2)'^'' 

i ° ^ (2-19) 

where is the convolution product of the functions X2 ^ ll(-^0i/i)("7 -''^2)|l2'-ti (gi) 
and X2 1-^ ll(-4/202)(-,-''^2)|l2rf2(gi)- It follows by Young's inequality again that 

l/mi 



^|L./.(g.) <{j IIM0jl)(-,^2)|IU(Bi)dX2) 

X (/ ||(^/.02)(-,X2)|rL.*.(gi)dX2) 



provided that mi, m2 S [1, 00] and + i;^ = + ^- For = ^, j = 1, 2, we get 

||^'||l=/'-(b2) < (II/i|Im;j'=i('^))''(II-^2|U/J2.=2(^))'', 

where we also used d^TTi]) . Then by and (ITTg)l we get 

I|w(/i,/2)||m-=(.#) < II/iIIm--w • II/2IIM--W, 

and this concludes the proof. □ 

Remark 2.23. A particularly sharp version of Theorem 12.231 holds for ri — si, 
r2 — 32, and r = s. That is, let r, ri,r2 G [1, 00] such that + -p^ = ^. It follows 
at once by Proposition 12.211 that for arbitrary /i, /2 G 7i-oo we have 

/2)||Mj'-(7r#) = ll/l|lA/;i'"i(7r) ' W M\ M^l'"'^ (tt) ^ 

which in turn implies that W{fi, /2) £ M^^{it'^) if and only if for j — \,2 we have 

Corollary 2.24. Let G he a simply connected, nilpotent Lie group with a unitary 
irreducible representation n: G ^ B{Ti), pick (j)i,4'2 S ^^oo \ {0}, o,nd denote $ = 
yV((/)i,^2) G 3(0). If r,ri,r2 E [l,oo] and ^ — ^ + then the cross-Wigner 
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distribution associated with any predual to the coadjoint orbit of the representation tt 
defines a continuous sesquilinear map 

Proof. One can apply Theorem 12.221 with ri = si, r2 = S2, and s = oo. Al- 
ternatively, a direct proof proceeds as follows. Let /i,/2 S 'H-oo- It follows by 
Proposition [2?2T] along with Holder's inequality that for every F S Se we have 

(w(/i,/2))(-,miL^(B.) < P^JiIIl-mb.) • UMl^h,.) 

whence ||W(/i, /2)||Mj=°(7r#) < ll/i IIm;!'"^! w ' II-^2|Im;2-'-2(^), and the conclusion 
follows. □ 

The next corollary provides a partial gener alization of Th. 4.3 in [^04] . 

Corollary 2.25. Let G be a simply connected, nilpotent Lie group with a unitary 
irreducible representation n: G ^ Biji), pick 0i,(/)2 G Ti-oa \ {0}, and denote $ = 
W'((/'i,02) G S{0). Assume that Qf, is a predual to the coadjoint orbit O associated 
with the representation tt, and let fle = Be + 0e ^^^2/ direct sum decomposition. If 
r, s, ri, si, r2, S2 G [l,oo] satisfy the conditions 

,1111,11 

r < s, ^2, S2 e [r, s\, and = = 1 , 

n r2 si S2 r s 

then the following assertions hold: 

(1) For every symbol a G Mj'*(7r'^) we have a bounded linear operator 

Op^a):M;i'^^n)^AQ-^^7:). 

(2) The linear mapping Op''(-): M^'{tt*) -> B{M^l'"' (n), AQ'"^ (n)) is con- 
tinuous. 

Proof. For every t G [l,oo] we are going to define t' G [l,oo] by the equation 
7 + 77 = 1. With this notation, the hypothesis implies — + ^ = — + 4- = ^ + ^ 
and moreover ri, si, S2 G [r', s']. Therefore we can apply Theorem l2.22l to obtain 

l|W(/2,/i)IU,.'V(,,) < II/i|Im--(.) • ll/2ll,,4.4(,. (2-20) 

whenever /i, /2 G H_oo- 

On the other hand, if a G M'^'^{ti'^), then 

(Op-(a)/i I /2) = (a I >V(/2,/i))l^(o) = (-Afa | (W(/2, /i)))l2(b,xb.) 

by CoroUarv 1 2 . 11 p ]) and Proposition 12. 8p ]). Then Holder's inequality for mixed- 
norm spaces (see for instance Lemma 11.1.2(b) in |Gr01| ) shows that 

l(Op'^(a)/i I /2)| < Mfa||i.,.(,^,,^) . (W(/2,/i))IL.',.'(,^,,^) 

= l|a|lM-'(.#)-||W(/2,/i)llM;'^='(.#) 

< l|a|lA/;-(^#) ■ II/i|Im;i-i(-) • ll/2llJ^,^-^-4(^^' 

where the latter inequality follows by (|2.20p . Now the assertion follows by a 
straightforward argument that uses the duality of the mixed-norm spaces (see 
Lemma 11.1.2(d) in [GrOT]). □ 
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Corollary 2.26. IfG be a simply connected, nilpotent Lie group with a unitary ir- 
reducible representation tt: G — > B{'H), then the following assertions hold whenever 

$ = W(01,02) With 01,02 e Hoc \{0}.- 

(1) For every a G M^'^(7r#) we have Op^(a) G B{n). 

(2) The linear mapping Op^(-): M^'^(7r'^) B{TC) is continuous. 

Proof. This is the special case of Corollary 12.251 with with ri = si = r2 — S2 — 2, 
r = 1, and s = oo, since Example EUT] shows that M'^^'^^n) ^H. □ 

We conclude this section by a sufRcient condition for a pseudo-differential opera- 
tor to belong to the trace class. In the special case of the Schrodinger representation 
of a Heisenberg group, a proof for this result can be found for instance in [Gr96j or 
|GH99j . 

Proposition 2.27. Let G be a simply connected, nilpotent Lie group with a unitary 
irreducible representation t:: G B{Ti), pick 01,02 G 'Woo with ||0i|| = ||02|| — 1? 
and denote $ = yV(0i, 02) G S{0). Then for every symbol a G M^' i'^'^) we have 
Op" (a) G 6i(H) and ||Op"(a)||i < ||a||^,ia(,#) . 

Proof. For arbitrary a G S'{0) we have by Corollarv l2.9tf T|) and Remark 12.201 
a = [f iA*a)iX,Y) ■ n* {expc^c{X,Y))'PdXdY, 



whence by Corollarv l2.11l 

Op" (a) = JJ {A*a){X, Y) ■ Op"(7r#(expG,G(^, Y))<^)dXdY (2.21) 

B=XBe 

where the latter integral is weakly convergent in £(7ioo, 'H-oo)- On the other hand, 
for arbitrary X, F G fle we get by (f2J3l) and CoroUarv [2lTpl) 

Op"(7r#(expG^G(^,n)<i') = ^(expG(^ + n) ° Op"(<i>) o 7r(expGX)-i 

= (• I 7r(expGX)02)^(expG(X + r))0i. 

In particular, Op"(7r#(expc.t,c.(X, r))$) G and 

||0p"(7r#(expG^G(^:n)<i')l|i = lk(expG(^ + n)0i|l ' lk(expG X)02|| - 1. 

It then follows that the integral in ()2.21|) is absolutely convergent in &i{'H) for 
a G (tt*) and moreover we have 



||Op"(a)||i < jj \{Ala){X,Y)\dXdY - ||a|U4.i(.#) 

which concludes the proof. □ 

3. The case of square-integrable representations 

In this section we focus on square-integrable representations of nilpotent Lie 
groups. A discussion of the crucial role of these representations along with many 
examples can be found for instance in the monograph [CG90j . 
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3.1. Independence of the modulation spaces on the window vectors. 

Lemma 3.1. Let Gi and G2 be unimodular Lie groups and assume that we have 
a group homomorphism a: Gi — > AutG'2, gi 1— > . Consider the semidirect 
product G = Gi X-a G2 and for every h G G and (j): G C define Rh(j)'- G —>■ C, 
{Rh4>){g) — (t>{gh). Fix r,sG [1,cxd] and consider the mixed-norm space L^''^{G) 
consisting of the equivalence classes of functions <j): G ^ 'C such that 

ll'/>IU-.'(G) := [j [J |?!)(<?i,52)rd5i) ^ d<72) ^ < 00, 

G2 Gi 

with the usual conventions if r or s is infinite. Then the space L^'^{G) is invariant 
under the right-translation operator Rh for every h E G, and the mapping 

p:G~^B(L-'\G)), h^R,\L-.^^G) 

is a strongly continuous representation of the Lie group G by isometrics on the 
Banach space L^'^{G). 

Proof. Let (f): G C be any measurable function and h = {hi, /12) G G. We have 
{Rh4>)i9i^ 92) — 0(51^1, ck/j-i (32)^2)- Since the group Gi is unimodular, it then 
follows that for every 172 S G2 we have 



|(^/i0)(5i,52)rdc/i = J |0(gi/ii,a^-i(52)/i2)rdc/i = J \4>{gi,a^-i{g2)h2)\''dgi. 

Gi Gi Gi 

Now, by integrating on G2 both extreme terms in above equality and taking into 
account that G2 is unimodular, we get 

(V/ieG) WRh^L^.s^G) = Uh^-HG)- 

With this equality at hand, it is straightforward to prove all the assertions in the 
statement just as in the classical case when r = s. □ 

Remark 3.2. In the setting of Lemma [5TT1 for every ip e L^(G) we can define the 
bounded linear operator p{'ip): i'''*(G) — * L'''*(G) by 



(Vx e L-''{G)) p(^)x = j Hh)RhXdh. 

G 

Then for every x G i'''*(G) we have 

{pi^)x){g) = / x{gh)Hh)dh for a.e. geG 



and 

\\pim\L^^HG)<c\mL^.iG), 

where G denotes the norm of the operator p{ip), hence G < \\iP\\l'^(g)- ^ 

We are now ready to prove a theorem that covers many cases when the modu- 
lation spaces for square-integrable representations do not depend on the choice of 
a window function. The second stage in the proof is inspired by the methods of 
the theory of coorbit spaces (see [FG88| . |FG89aj . |FG89b| . and also the proof of 
Prop. n.3.2(c) in [GrOl]). 
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Theorem 3.3. Let Gi and G2 be simply connected, nilpotent Lie groups and a 
unipotent homomorphism a : Gi — > Aut G2 ■ Define G = Gi K „ G2 and assume that 
the center I of q satisfies the condition 

3 = (3 n 01) + (3002). (3.1) 

Assume the irreducible representation tt: G — > B(Ti.) is square integrable modulo the 
center of G, and pick any Jordan-Holder basis in such that for the corresponding 
predual Qe for the coadjoint orbit associated with tt we have ge = (flen0i) + (0enfl2)- 
Then the modulation spaces for the representation n with respect to the decom- 
position 2e — (fle Hfli) X (fle (^32) 0,^6 independent on the choice of a window vector 

Proof. The proof has two stages. 

1° For the sake of simphcity let us identify the Lie group Gj to its Lie algebra Qj 
by means of the exponential map expg.^. , so that Gj will be just Qj with the group 
operation * defined by the Baker-Campbell-Hausdorff series. Let Z be the center 
of G, whose Lie algebra 3 is the center of fl. Then we have a linear isomorphism 
Qe — q/}, X ^ X -\- and we shall endow Qe with the Lie algebra structure which 
makes this map into an isomorphism of Lie algebras. 

If we define Ge '■— G/Z, then Ge is a connected, simply connected nilpotent Lie 
group, whose Lie algebra is just fle- Let *e denote the multiplication in Ge, which 
is just the Baker-Campbell-Hausdorff multiplication in Qe- 

Now use assumption (13. ip to see that if (11,12) G 3 Q = fli Xq fl2, then 
(Yi, 0), (0, 12) G 3. Now formula (|2.2p shows that for every {Xi,X2) G fl we have 

= [{Xi,X2), (Yi, 0)] = ([Xi, Fl], -d(Yi)X2), hence Yi belongs to the center 31 of 

01 and a(li) = 0. This shows that the closed subgroup Zi :— Z Ci Gi is contained 
in the center of Gi and satisfies 

ZiCKera. (3.2) 

Also = [(^1,^2), (0, 12)] = (0, a{Xi)Y2 + [X2, Y2]) for every (^1,^2) G 0, whence 
we see that I2 belongs both to the center 32 of G2 and to Ker {a{Xi)) for arbitrary 
Xi G fli. Therefore the closed subgroup Z2 := Z H G2 is contained in the center of 
G2 and we have 

(VgiGGi) ag,{Z2)CZ2. (3.3) 
It follows by ()3.2|) and (j3.3p that the group homomorphism a : Gi ^ Aut G2 induces 
a group homomorphism a: Gi/Zi — > Aut {G2/Z2) and we have the isomorphisms 
of Lie groups 

Ge ~G/Z~ (Gi/Zi) Kg(G2/Z2). 

Moreover Z ~ Z\ x Z2. 

2° We now come back to the proof. Fix r, s G [1, cxd] and let 0i, (/'2 G Hoo be any 
window functions with ||0i|| ~ ||02|| = 1- For = 1, 2 and every / G 7i-oo we have 
by Corollary EH 

(^^,/)(X) = (/|7r(expGX)02) 

= y x(r)(7r(expcr)<^i |^(expGX)02)dr 

fle 

- y x{y){4>i |^(expG((-y)*x))02)dr 

fle 
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/ 



x{Y)e' 



I 



I 



Be 



for every X e Qe, where x '■= -^(pif G L^^'^iSe x 0e) and a : x gg — > R is a suitable 
polynomial function defined in terms of the central character of the representation tt 
(see e.g., jMa07| V Now note that A^^cj)! G S{ge) by CoroUarvl^^gp)) . It then follows 
by Lemma 13.11 and Remark 13.21 that there exists a constant C > such that for 
every / e H-oo we have M^a/IU'-.^lBeXB,) < M0i/||L'-.=(fleXfle)- Thus we get 
the continuous inclusion map M^'^{tt) ^ M^^{n). Now the conclusion follows by 
interchanging (pi and 4>2- D 

The previous theorem allows us to omit the window vector in the notation for 
modulation spaces associated to square- integrable representations. 

Example 3.4. Theorem 13 . 31 applies to a wide variety of situations. Let us mention 
here just a few of them: 

(1) In the case of the Schrodinger representation of the Heisenberg group 
El2n+i = IX K"^^ we recover the well-known property that the classical 
modulation spaces used in the time-frequency analysis are independent on 
the choice of a window function (see for instance Prop. 11.3.2(c) in [GrOlj ). 

(2) We shall see below (see subsection 13.31) that one can give sufficient condi- 
tions for the continuity of the operators constructed by the Weyl-Pedersen 
calculus for the square-integrable representation n: G B{Ti.) by using 
spaces of symbols which are modulation spaces M^'''{tt'^) C S'{0). Here 
tt"^ : G K G B{L'^{0)) is in turn a square integrable representation to 
which Theorem 13.31 applies and ensures that the corresponding modulation 
spaces do not depend on the choice of a window function. 



3.2. Covariance properties of the Weyl-Pedersen calculus. We now record 
the covariance property for the cross- Wigner distributions and its consequence for 
the Weyl-Pedersen calculus. In the very special case of the Schrodinger represen- 
tation for the Heisenberg group we recover a classical fact (see e.g., |Gr01) ). 

Theorem 3.5. Assume that the representation tt: G ^ B{Ti.) associated with O is 
square integrable modulo the center of G. Then the following assertions hold: 

(1) For every f,h £ Ti. and X Cz Q we have 



□ 




) for a.e. ^ G C 



(3.4) 
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(V^gO) ^b-^oL (3.5) 
CoIb. = 0. (3.6) 
Also, it easily follows by Definition 12.71 that for arbitrary f,h £ H we have 

^if,h){0 = J c'<«-^>(/ I n{exp(jX)h)dX for a.e. ^eO. 

Be 

It then follows that for arbitrary Xq G Qe and a.e. ^ G O we have 
>V(^(expG Xo)/,^(expG Xo)/i)(0 

e'<«^^>(/ I 7r((expG(-Xo))(expGX)(expGXo))/i)dX 



Be 



J e'<«'^>(/ I 7r(expG(e"'^«(-^")X));i)dX 



If we denote by pr^ : g ^ 3 the natural projection corresponding to the direct sum 
decomposition g — i + Qe, then we have for every X € g^, 

where we have endowed Qe with the Lie algebra structure which makes the lin- 
ear isomorphism 0e — &/} into an isomorphism of Lie algebras (see also [Ma07j ). 
Therefore, by using (|3.4p and p.6p . we get 

(VX e ge) 7r((expG(e"^»(-^«)X))) = e'<«'''°°'''"''°'^>^(expG(e"'^«=(-^'')x)) 
and then the above computation leads to 
W(^(expG Xo)/,^(expG Xo)hm 

Be 

\i{i,x)^-H(o--^i--o^r^o,x)^f I ^(^e^p^ie-<i.J-Xo)x))h)dX 

Be 

'■^i(«.c-»eXo^)^_i((,.d,(-Xo,).^^^^.a,,Xo^^^^ I ^(expGr)/l)dy 

Be 

where we used the change of variables X i-^Y = e'^'^^^^~'^°^ X , which is a measure- 
preserving diffeomorphism since ge is a nilpotent Lie algebra. Now note that by 
using p.Sp we get for a.e. ^ G O and every F G ge, 

= (e,e"'^^^«y) - {^,pr^{e'^o^"Y)) 

- {£^o,c'''^^^-^"\c'''^o^"Y - pr^{e'"^o^°Y})) 

= {^,e'"^»^°Y) 
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since (^o , i^} = by (13. 6|) . Thus the eonclusion foUows by the formula we had 
obtained above for W(7r(expQ Xo)f, 7r(expQ Xo)h){£^), and this completes the proof 
for X G Qe- Then the formula extends to arbitrary X G g by using the fact that 
= 02+3 and taking into account (|3.4p . 

(HI If a e S{0), then for every /, e 7i we have 

(Op(a o Ad^(5-i)|o)0 I f)H = (a o Ad*a{g^')\o \ cb))mo) 

= {a\W{f,^)oAd*aig)\o)LHo) 
= (a| W(^(<?)-V,^(<?)"V))l^(0) 
= (Op(a)^(5)-V I 
= (7r(g)Op(a)^(5)-V | f)n, 

where the first and the fourth equalities follow by Corollary l2.1Hf T|) , the second 
equality is a consequence of the fact that the coadjoint action preserves the Liouville 
measure on O, while the third equality follows by Assertion ([T|) which we already 
proved. 

Thus we obtained the conclusion for a G S{0), and then it can be easily extended 
by duality to any a G S'{0) by using equation (|2.10p in Definition 12. 101 □ 

3.3. Continuity of Weyl-Pedersen calculus. 

Lemma 3.6. Let G be any Lie group with a unitary irreducible representation 
TT-.G^ B{n) and 

7f : G K G ^ B{62{H)), Tt{g, h)T = TT{gh)TTi{g)-^ . 

If G is a unimodular group and tt is square integrable modulo the center of G, then 
n is square integrable modulo the center ofGtKG. 

Proof. If TT is square integrable modulo the center Z of G, then there is 0o G H\{0} 
such that the function gZ i— > \iT^ig)4'o I 4'o)\ is square integrable on G/Z. Let us 
define the rank-one projection Tq = {■ \ (j)o)(t>o Then we have 

\{7f{g,h)To\To)\'dgdh= J J |(7r(5/i)To^(g)-i | To)pd/i)d.g 

(GkG)/(ZxZ) G/Z G/Z 

- I if MhWoAg)-' I To)\'dh)dg. 

G/Z G/Z 

Since Tq = (■ | (/'o)0o, we get 7r(/i)To7r(5)"^ = (• | 7r(g)(/io)7r(/i)0o, and then 
{7rih)To7rig)-' \ To) = (^(/i)0o | M ' i<t>o I ^(5)0o). 

Therefore 

\{n{g,h)To I To)\'dgdh = ( J |(^(.g)(/.o | Ml^dh)^ 

{GkG)/(ZxZ) G/Z 

hence the function {g,h){Z x Z) i-^ \{TT{g, h)To \ To)\ is square integrable on the 
quotient group {G K G)/{Z x Z), and this concludes the proof since Z x Z is the 
center of G ix G (see Example I2.3p . □ 
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Remark 3.7. Assume that n: G ^ B{H) is a square-integrable representation of 
a simply connected, nilpotent Lie group, with the corresponding coadjoint orbit 
O C 0*. Recah the representation 7r# : G x G B{L'^{0)) from Definition [m] 
(see also Remark 12.201) . The assumption that tt is square integrable modulo the 
center of G implies by Theorem I3.5t (2|) that tt^ is given by 

^# : G X G ^ BiL^O)), n*{g, expY)f = (e'<-^>#/) o Ad*aig~')\o- 

Since the unitary operator Op'^ : L'^{0) &2{'H) intertwines tt* and tt, we get 
by Lemma 13.61 that tt"^ is also a unitary irreducible representation which is square 
integrable modulo the center Z x Z of G k G. □ 

Corollary 3.8. Let G be a simply connected, nilpotent Lie group with a unitary ir- 
reducible representation tt : G — > B(Ti.) which is square integrable modulo the center 
ofG. Ifr,ri,r2 & [l,c»] and ^ = + then the cross- Wigner distribution asso- 
ciated with any predual to the coadjoint of the representation tt defines a continuous 
sesquilinear map 

Proof. Firstly use Corollarv l2.24l Then the conclusion follows since both tt and tt"^ 
are square integrable representations (see also Remark 13. 7p , hence Theorem 13.31 
shows that the topologies of the modulation spaces Af^-''^ (tt), Af^'''^ (tt), and 
M'''°°(7r^) can be defined by any special choice of window functions. □ 

Corollary 3.9. If G be a simply connected, nilpotent Lie group with a unitary ir- 
reducible representation n: G ^ B{H) which is square integrable modulo the center 
ofG, then the cross- Wigner distribution associated with any predual to the coadjoint 
of the representation it defines a continuous sesquilinear map 

Proof. Just apply Corollary 13.81 with ri — r2 — 2 and r — 1; and recall from 
Example [2T7] that M'^''^iTr) = H. □ 

In the special case of the Schrodinger representation for the Heisenberg group, the 
following corollary recovers the assertion of Th. 1.1 in |GH99| concerning the bound- 
edness of pseudo-differential operators defined by the classical Weyl-Hormander 
calculus on W\ 

Corollary 3.10. Let G be a simply connected, nilpotent Lie group with a unitary 
irreducible representation tt : G — > B{Ti) which is square integrable modulo the cen- 
ter ofG. If r,r' ,ri,r2 S [l,c>o] satisfy the equations ■^ = ^ + ^ = i — -p, then the 
following assertions hold: 

(1) For every symbol a G Af '"'^(vr'^) we have a bounded linear operator 

Op'' (a): AFi'''i(7r) ^ Af2'''2(7r). 

(2) The linear mapping Op''(-): IVr'^iTT*) S(Ari^''i (tt), Af^^''^^ (vr)) is con- 
tinuous. 

Proof. Firstly use Corollarv 12.251 Then the conclusion follows since Theorem 13.31 
shows that the topologies of the modulation spaces involved in the statement can 
be defined by any special choice of window functions. □ 
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Corollary 3.11. IfG be a simply connected, nilpotent Lie group with a unitary ir- 
reducible representation vr: G — ^ B(TC) which is square integrable modulo the center 
of G, then the following assertions hold: 

(1) For every a G A/°°^i(7r#) we have Op'' (a) £ B{H). 

(2) The linear mapping Op''(-): M°°'^{tt'^) B{TC) is continuous. 

Proof. This is the special case of Corollary 13. 101 with with ri — r2 — 2 and r — 1, 
since Example [2Tfl shows that M2'2(7r) =71. □ 



4. SCHRODINGER REPRESENTATIONS OF THE HeISENBERG GROUPS 

We show in the present section that, in the special case of the Heisenberg group, 
the modulation spaces of symbols defined in our paper are in fact nothing else than 
the modulation spaces widely used in time-frequency analysis. 

Schrodinger representations. Let V be a finite-dimensional vector space en- 
dowed with a nondegenerate bilinear form denoted by (p, q) ^ p ■ q. The corre- 
sponding Heisenberg algebra f)v = V x V x M is the Lie algebra with the bracket 

[{q,p,t),[q\p',t')] = [(0,0,p-g'-p'-g)]. 

The Heisenberg group Hy is just f)v thought of as a group with the multiplication * 
defined by 

X *Y = X + Y + ]^[X,Y]. 

The unit element is G Hy and the inversion mapping given by X~^ := —X. 
The Schrodinger representation is the unitary representation vry : Hy B{L'^{V)) 
defined by 

(7rv(g,p,i)/)(a;) =e'(f^+^f9+*)/(a; + 9) for a.e. x e V (4.1) 

for arbitrary / g i^(V) and {q,p,t) G Hy. This is a square-integrable representa- 
tion and the corresponding coadjoint orbit of Hy is 

O = {^: i)y ^ R linear | ^(0, 0, 1) = 1}. (4.2) 

Let ^0 G C be the functional satisfying ^o{q,p,0) = for every q,p G V. If we 
denote dimV = n, then any basis {xi,...,a;„} in V naturally gives rise to the 
Jordan-Holder basis 

(a;i, 0, 0), . . . , {x„,0, 0), (0, xi, 0), . . . , (0, 0), (0, 0, 1) 

in f)y and the corresponding predual of O is 

(f)y), = V X V X {0}. 

For the sake of an easier comparison with the previously obtained results we shall 
denote G = Hy and = f)y from now on, and in particular we shall denote 
Be = (f)v)e. 
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Computing the Moyal product representation. Recall from [Ma07| that for 
every f,hE S{0) we have 

(Ve e O) = JJ e'<«'^+^>e('/2)<««'[^'^l>7(X)/J(r)dXdy. 

Bexge 

It then follows by a duality argument that for every / G S{0) and V E q we have 
(/#c-'<-'^>)(0 = J e'<«-^-^>e('/2)<«-[^'-^l)/(X)dX, 

So 

whence 

(Vye0)(VeeO) (/#e-'<-'^>)(0=e-'<«'^>/(^ + (l/2)Cooadgl^). (4.3) 
Since f^fh = h^f, we also get 

iWegmeO) (e'<-'^>#/)(e) = e'<«'^>/(^ + (1/2)^0 o ad,y). (4.4) 
Now for arbitrary X,Y E q, f E ^{0), and ^ G O we get 

(e'<-^^+^>#/#c-'<-^>) =e'<«^^+^)(/#e-'<-^))(e + (1/2)^0 o ad,(X + Y)) 

^^i{!i,X+Y)^-i{i+{l/2)^ooM,{X+Y).X) 

X /(e + (1/2)^0 o adg(X + Y) + (1/2)^0 o ad^X) 

=e'<«^^>e('/2){€oJxy]> 

x/(e+ (1/2)^0 oadg(X + (l/2)y)). 

By taking into account (j2.13p . we now see that the unitary irreducible representation 
tt*:GkG^ B{L^{0)) is given by 

(7r#(expG,G(^, Y))fm - e'««-^)+<«o-[^-^l)/2)/(C + o adg(X + (l/2)y)) (4.5) 
where the latter equation follows by (|4.2p . 

Abstract unitary equivalence. Denote the center of G by Z, with the corre- 
sponding Lie algebra 3. The above formula yields expg^g(3 x {0}) C Ker tt* , hence 

we get a unitary irreducible representation TTy : (G k G)/(Z x {1}) B{L^{0)). 
Also note that there exist the natural isomorphisms of Lie groups 

(G K G)/iZ X {1}) ~ (G/Z) K G ~ Hvxv- (4.6) 

By specializing (|4.5p for X, F e 3 we can see that the representation TTy has the 
same central character as the Schrodinger representation of the Heisenberg group 
HvxVi hence they are unitarily equivalent to each other, as a consequence of the 
Stone-von Neumann theorem. 

Specific unitary equivalence. Alternatively, we can exhibit an explicit unitary 
equivalence as follows. Let us consider the afiine isomorphism O — > (V x V)*, 
£. ^ ClvxVx{o}: and the natural embedding VxV~VxVx {0} ^ [)v- Now for 
X,y e V X V audi G R we have iX,Y,t) e Hyxv ^ (G k G)/(Zx {1}) (see (ITO ) 
hence 

(4 (exPHv X V ^' ^) )/) (^) = ('^v (exPG K G ( 0) , (y, t) ) ) /) (0 

^gi««,Y)+t+^,„(x,y)/2)^(^ ^ ^ ^^^^X + (l/2)y)) 
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where uj^g{V, W) := {£_q, [V, W]) whenever V,W E V x V ^ g. Thence we get 

(4(expH^^^ {X {1/2)Y, Y, t))m) = c'««'^>+*+-«o(^.^)/2)/(^ + ^„ „ ad,X). 

Note that tp: i)vxv fivxv, {X,Y,t) i-^ {X — {l/2)Y,Y,t) is an automorphism 
of the Heisenberg algebra l)vxv, hence, by denoting by Vt: Hyxv — > IHIvxV the 
corresponding automorphism of the Heisenberg group HvxVj we get 

(^(expH,., Y, t))fm = e'««^^)+-^o(^^^)/2+*)/(^ + ^„ o ad,X) 

where tt :— TTy o v]/ is again a representation of the Heisenberg group Hyxv- Then 
for arbitrary G V x V we get 

(^(expH^.v (X, Y, t))mo + eo o ad,y) ^e'(-.o(^.>')+-.o(^.>')/2+*) 

X fi^o+^oo^d,{V + X)). 

Now let us define the affine isomorphism 

A:VxV^O, V ^ £,0 + Co o adgV 

and consider the unitary operator U : L^(V x V) ^ L?{0)^ f ^ I ° A^^. It follows 
by the above equation that if we define the Heisenberg group Hyxv by using the 
nondegenerate bilinear map 

(V X V) X (V X V) ^ M, {V,W) ^ -uj^g{V,W), 

then the unitary operator U intertwines the representation tt: Hyxv — * 'S(L^(C')) 
and the Schrodinger representation ttvxv ■ Hyxv ^ B{L^{V x V)). In other words, 
the operator U induces a unitary equivalence of the representation TTy with the 
representation ttvxv ° 

Determining the modulation spaces of symbols. It follows by the above dis- 
cussion that the operator U induces isomorphisms between the modulation spaces 
for the representations 

tt*:GkG^B{L'^{0)) and vryxv o ^""^ : Hyx v ^ ^(^^(V x V)). 

Now note that for arbitrary r, s G [1, oo] we have M''''*(7rvxv ° '^^^) = M'^'^{'!T\!xv) 
since the norm of any measurable function / : V x V ^ C in L''''*(V x V) is equal 
to the norm of the function {X, Y) i-^ f{X + {1/2)Y, Y) in the same space. There- 
fore the operator f ^ f o A^^ actually induces an isomorphism from the modu- 
lation space Af'''''(7ry ) onto the modulation space M'''*(7rvxv) of the Schrodinger 
representation TTyxv^ IHvxv B{L?{V x V)). Finally, recall that M''''*(7rvxv) — 
M'''*(V X V), where the latter is just the classical modulation space on V x V as 
used for instance in [GrOlj . 
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